Control using movements

ABSTRACT

A movement of an object is recognised as a predetermined movement, by transmitting signals between transmitter-receiver pairs, which are reflected from the object. A first event is recorded for one of the transmitter-receiver pairs if a reflected signal meets a predetermined proximity criterion, and a second event is recorded for a second transmitter-receiver pair if after the first event, a subsequent reflected signal meets a predetermined proximity criterion. The first and second events are used to identify the movement.

RELATED APPLICATIONS

This application is the National Stage of International Application No.PCT/GB2010/001215, filed Jun. 21, 2010, which claims the benefit of GB1005345.2, filed Mar. 30, 2010, and GB 0911840.7, filed Jul. 7, 2009.Each of these applications is hereby expressly incorporated by referencein its entirety herein.

BACKGROUND OF THE TECHNOLOGY

This invention relates to methods and apparatus for identifying apredetermined movement of an object; in particular, in some embodiments,for identifying a gesture by a human hand for the purposes of providinga gesture-based user interface to an electronic device.

It is known to track the movement of an object, such as a user's fingeror hand, by transmitting a succession of signals (e.g. ultrasoundpulses) from one or more transmitters, receiving reflected signals atone or more receivers, and tracking movement of one or more objects byanalysing changes in the received signals over time. It has beenproposed to apply such technology to user interfaces for electronicdevices, enabling, for example, a finger tip or hand to be moved inorder to control an on-screen cursor. Arrangements similar to this aredescribed in U.S. Pat. No. 5,059,959 (Barry) and U.S. Pat. No. 6,313,825(Gilbert).

Such an approach, however, requires a high level of computationalprocessing in order to track the object. This need for dedicatedprocessing and electrical power is undesirable; particularly so in thecontext of resource-constrained mobile devices. The Applicant hasrealised that such a resource-intensive tracking approach is notnecessary in some circumstances, particularly when the movements ofinterest take place close to a screen or other defined surface.Furthermore, conventional tracking methods may have difficulty indiscerning the direction of an object using baseline time-of-flight orarray based methods due to the speed at which the object moves, and itscontinually changing ‘shape’ relative to the transducer setup.Conventional tracking methods also rely on there being a clear pointreflector or a clear front to track.

SUMMARY OF CERTAIN INVENTIVE ASPECTS

When viewed from a first aspect, the invention provides a method ofrecognising a movement of an object as one of a set of one or morepossible predetermined movements, the method comprising:

-   -   continuously or repeatedly transmitting signals between at least        two transmitter-receiver pairs, said signals being reflected        from said object to produce reflected signals;    -   recording a first event for a first one of said        transmitter-receiver pairs if a said reflected signal meets a        predetermined proximity criterion;    -   recording a second event for a second one of said        transmitter-receiver pairs if, after the first event, a        subsequent reflected signal meets the or a further predetermined        proximity criterion; and    -   using said first and second events to identify said movement as        the or one of the predetermined movements.

The invention extends to apparatus for recognising a movement of anobject as one of a set of one or more possible predetermined movements,comprising at least two transmitter-receiver pairs; and an electroniccontroller arranged to:

-   -   continuously or repeatedly transmit signals between said at        least two transmitter-receiver pairs, said signals being        reflected from said object to produce reflected signals;    -   record a first event for a first one of said        transmitter-receiver pairs when a said reflected signal meets a        predetermined proximity criterion;    -   record a second event for a second one of said        transmitter-receiver pairs when, after the first event, a        subsequent reflected signal meets the or a further predetermined        proximity criterion; and    -   electronic processing means arranged to use said first and        second events to identify said movement as the or one of the        predetermined movements.

The invention also extends to a computer software product, and to acarrier bearing the same, configured, when run on a computer, torecognise a movement of an object as one of a set of one or morepossible predetermined movements, the software comprising logic arrangedto:

-   -   receive data resulting from continuously or repeatedly        transmitting signals between at least two transmitter-receiver        pairs, said signals being reflected from said object to produce        reflected signals;    -   recording a first event for a first one of said        transmitter-receiver pairs if said data represents a said        reflected signal meeting a predetermined proximity criterion;    -   recording a second event for a second one of said        transmitter-receiver pairs if, after the first event, said data        represents a subsequent reflected signal meeting the or a        further predetermined proximity criterion; and    -   using said first and second events to identify said movement as        the or one of the predetermined movements.

The above-mentioned predetermined criterion may be a first predeterminedproximity criterion with the above-mentioned further predeterminedcriterion being a second predetermined proximity criterion

Thus it will be seen by those skilled in the art that, in accordancewith the invention, certain movements of an object, conveniently a humanhand, can be recognised. However rather than usingcomputationally-expensive and technically-difficult precise tracking ofthe location of the object in three-dimensional space, such recognitionis made by detecting the object with two different transmitter-receiverpairs at different times. The Applicant has appreciated that this,together with knowledge of the placement of the transducers of twotransmitter-receiver pairs, can be used to identify the movement. Forexample the order in which the first and second events occur may be usedto recognise a sweeping gesture from left to right across the face of ascreen such as a computer display. This could then be used e.g. toeffect a scroll or other sideways movement of something displayed on thescreen.

The proposed event-based gesture recognition method does not need theassumption of a clear point reflector or clear front to track. Insteadit is based on two events being detected in two similar but spacedchannels; or two channels more generally. In fact, the details of thereflected signals could be hard to identify and interpret even for askilled observer, but could nevertheless be used for precise gesturerecognition using the proposed invention.

The predetermined set of movements could be a discrete set or acontinuous set.

The proximity criteria could comprise or make use of instantaneoussignals, i.e. simply those received in a single given time-slot. Howeverthey are not limited to this and could equally be based on signals overa number of, e.g. adjacent timeslots; for example the criteria couldcomprise an average, or a gradient or other trend.

In the preferred set of embodiments the transmitter and receiver of eachpair are separated from one another across at least part of a controlsurface. In some embodiments this could be a passive area designated forrecognising movements performed over it, but in many advantageousembodiments the control surface comprises a display for an electronicdevice such as a computer so as to allow interaction with the devicethrough the movements recognised.

Preferably for each pair, the transmitter is on an opposite side of thecontrol surface to the receiver. In general in accordance with theinvention it is preferred to employ discrete transducers that areseparated from one another as opposed to being provided in an array (inthe sense in which that term is understood by those skilled in the art).

The Applicant has found that this approach is most useful when theobject is moving close to the control surface. The predeterminedproximity criterion/criteria can be used to ensure this. It/they couldcomprise a maximum time of flight of the acoustic signal from thetransmitter to the receiver via the object; that is, a maximum bound onthe sum of the distance from the transmitter of a transmitter-receiverpair to the object and the distance from the object to the receiver ofthe transmitter-receiver pair. Such a bound effectively defines anellipsoid boundary for a transmitter-receiver pair within which theobject must lie in order to satisfy the proximity criterion for thattransmitter-receiver pair. In some embodiments the proximity criteriacould be defined relative to the control surface.

Additionally or alternatively, the proximity criterion/criteria couldcomprise a minimum energy reflected from the object. It will be seenthat both of this and the time-of-flight technique effectively give ameasure of how close the object is to the screen or other controlsurface (if such is present). The invention includes any other techniquefor determining such proximity.

Having the proximity criteria configured such that only gestures closeto a screen are detected is beneficial in the context of a computer,particularly a laptop, since false detections from typing on a keyboardcan more easily be avoided.

Whilst not essential, it is preferred that the signals are acoustic,e.g. ultrasonic signals.

Where a control surface such as a display screen is provided andmovements need to be recognized close to that surface, the Applicant hasappreciated an advantage of using acoustic signals, namely that by usingomni-directional or partially omni-directional transmitters andreceivers it is not necessary to have them projecting out of the planeof the screen or other surface. Rather, one or more of them can bemounted flush with or below the surface. This is significant from anaesthetic design point of view.

Simplistically and typically the proximity criterion for recording thefirst and second events would be the same. However this is notessential. For example the criterion could differ between thetransmitter-receiver pairs. This would amount to the ‘sensitivity zone’being asymmetric which might be desirable in some applications. Equallythe criteria could differ between the first and second event recordal.For example the first event may require the object to be closer than forthe second event to minimise false detections.

As the first and second events relate to signals transmitted atdifferent times (either different discrete transmissions or differentportions of a continuous transmission) they do not simply correspond todifferent perspectives of the object when it has not significantly movedbut are triggered only because the object is moving. In the simplestembodiments the events simply give a binary indication of whether or notthe object is in a detection zone defined by the positions of thetransmitter and receiver and the proximity criterion. However the orderin which the object passes between two or more of these zones can beusefully used to detect movements.

In accordance with the invention the first event might be recordedbefore receipt of the signals transmitted at a later time and which giverise to the second event. However this is not essential nor indeedtypical. More typically, unless the movement of the object is very slow,the first event is only identified and recorded after the signalscorresponding to both events have been received. Once the first eventhas been recorded, the received signals (which are stored in a buffer orother storage) can be analysed to determine and record the second eventand so identify the gesture.

In preferred embodiments a quantitative measure of the proximity of theobject to the respective transmitter-receiver pairs is made. Comparingthe quantitative measures allows the first and/or second events to berecorded in respect of the transmitter-receiver pair closest to theobject—i.e. which has the highest proximity measure. This allows thedetection zones referred to above to overlap; in which case thequantitative comparison amounts to a determination of which zone theobject is closest to the centre of. For example where time-of-flight isused to determine proximity, if the respective ellipsoids defined by themaximum time-of-flight condition overlap, the quantitative comparison isequivalent to defining a virtual boundary surface through theintersection region of the ellipsoids, with the object causing an eventto be recorded for one pair if it is on one side of the virtualboundary, and an event for a different pair if it is on the other side.

The first and second events may be used in any appropriate way toidentify the movement as the or one of the predetermined movements. In aset of embodiments, only the identity of the transmitter-receiver pairscorresponding to the first and second events (their locations of coursebeing known) and the order in which they occur are used. However, inother embodiments, other factors such as the time interval between thefirst and second events (a measure of the speed with which the movementis executed) and/or the quantitative measure mentioned abovecorresponding to each of the events may be used.

The idea of recognising a movement depending on the identity oftransmitter-receiver pairs satisfying predetermined criteria forreflected signals is new and inventive in its own right and thus, whenviewed from a second aspect, the invention provides a method forrecognising a movement of an object as one of a set of one or morepossible predetermined movements, said method comprising determining anacoustic signal travelling between a first transmitter-receiver pair viaan object to meet a first predetermined proximity criterion; determininga further acoustic signal travelling between a secondtransmitter-receiver pair via said object to meet a second predeterminedproximity criterion; returning an indication that the, or one of the,predetermined movement(s) has taken place depending on whichtransmitters and receivers form the respective first and secondtransmitter-receiver pairs.

This aspect extends to apparatus for recognising a movement of an objectas one of a set of one or more possible predetermined movements,comprising processing means configured to:

-   -   determine that an acoustic signal travelling between a first        transmitter-receiver pair via an object meets a first        predetermined proximity criterion;    -   determine that a further acoustic signal travelling between a        second transmitter-receiver pair via said object meets a second        predetermined proximity criterion; and    -   return an indication that the predetermined movement, or one of        the predetermined movements, has taken place depending on which        transmitters and receivers form the respective first and second        transmitter-receiver pairs.

In any aspect of the invention, processing means may comprise one ormore computer processors, or silicon chips, such as an ASIC or FPGA. Itmay comprise a plurality of separate processors, connected bycommunication links.

In a set of embodiments the first and/or second predetermined proximitycriteria comprise a maximum time of flight of the acoustic signal fromthe transmitter to the receiver via the object. One or both mayadditionally or alternatively comprise a minimum energy reflected fromsaid object. In some embodiments, the first criterion is the same as thesecond criterion.

As for the previous aspects of the invention, in some embodiments, theproximity criterion comprises a trend e.g. a maximum or minimum rate ofchange in a value, or simply the sign of the change—positive ornegative. This could be an increase or decrease of reflected energy overa period of time, or a trend of increasing or decreasing of proximityover a period of time.

Other preferred features of the previous aspect may also be preferredfeatures of this aspect.

When viewed from another aspect, the invention provides a method ofrecognising a gesture comprising using the order in which detection ofan object is made by different pairs of acoustic transmitters andreceivers.

This extends to apparatus comprising a plurality of transmitter-receiverpairs and processing means configured to recognise a gesture using theorder in which detection of an object is made by the different pairs.

Preferably the transmitter and receiver of each pair lie across at leastpart of a predetermined active region.

In some embodiments the detection uses time of flight of the signal.Additionally or alternatively it uses energy reflected by the object.Additionally or alternatively, it uses time of flight and/or energy formultiple echoes or echo profiles.

The use of multiple echoes or a full echo-profile has importantconsequences. In many standard applications, ultrasonic ranging sensorsare used as “single-range” measurement devices. For example, theSensComp 600 Series Smart Sensor from SensComp, Inc. of Michigan, USwill output a single range to an object, such as a wall or a person. Incarrying out this distance measurement, however, a series of receivedechoic signal are analyzed, and only one is detected as stemming fromthe object of interest. From this perspective, such an ultrasonicranging device is similar to an infra-red ranging system, in that asingle number readout is provided. In doing so however, the other echoictaps, i.e. the rest of the echoic profile is discarded. For applicationssuch as the ones described herein, the full echo profile, or parts ofit, may be very important. Using such profiles, one can for instanceprocess multiple objects located at multiple ranges. Importantly, itallows the question of “where is the object, and is it within a certainrange interval” to be replaced by the question “is there an objectwithin this specific range interval”. The first question can be seen tobe one favouring selection before measurement. The second allowsmeasurement without selection. In practice, this means that for instancea proximity criterion, such as outlined elsewhere in this application,can be set to detect whether there is an object present within a certainrange interval, i.e. by using an energy detector, an edge detector, or aenvelope shape detector applied to a certain portion of the returnsignal, or an impulse response. This detector could be oblivious as towhether there are other objects at other ranges, some of which wouldnormally trigger an off-the-shelf ultrasonic ranging system.

In contrast, single-sensor or few-sensor infrared systems do not havethe ability to process echoic profiles, and in detecting a most likelyrange for an object as it moves along a path, may have large variancesin the readouts during the movement. In the event that an object, suchas a hand, is moved in a non-rigid fashion, such as a scoop, an infraredsingle-ranging sensing system is “faced with the dilemma” of reportingthe distance to the fingertips, which are moving, or to the wrist, whichis static. Due to the fast speed of light, the reflections stemming fromthose two objects cannot be resolved while keeping a sufficiently wideopening angle, such as is required to capture the movement of a hand. Asa consequence, infrared sensing systems facing this ‘range-smearing’problem may favour the rigid motion of an object past sensors. Thiscould be an extended hand, or a finger extended out from a hand of whichthe palm is parallel with the surface to which the sensors are mounted.

Typically, by working with parts of the echo profile as opposed to asingle range, the ultrasonic solution does not have this problem. Aswill be seen, acoustic, especially ultrasonic arrangements describedherein in accordance with the invention may effectively ‘reuse’ the datathat is normally wasted or thrown out in typical echo-location systems,and which is unavailable in infrared systems due to range-smearing.

Where reference is made herein to “time-of-flight” measurements, thisshould be understood to include a time-of-flight measurement extractedfrom a plurality of possible candidates given by the echoic profile.

When viewed from a further aspect, the invention provides a method ofidentifying motion of an object from a first region to a second region,comprising:

-   -   using a first acoustic signal to determine that, at a first        time, the object is located in said first region;    -   using a second acoustic signal to determine that, at a second,        later time, the object is located in the second region;    -   storing the results of said determinations in an electronic        register or memory; and    -   identifying motion of the object from the first region to the        second region if the results of said first and second        determinations are both positive.

Although in accordance with the invention at least twotransmitter-receiver pairs are provided, this does not necessarily implythat they are fully independent, i.e. with one transmitter and onereceiver for use solely by that pair. For example the two pairs couldshare a common transmitter or a common receiver to reduce the number ofparts required. A single transducer may be able to perform the functionof both a transmitter and a receiver; in such cases the transducer wouldtypically act as the transmitter of one transmitter-receiver pair andthe receiver of a different transmitter-receiver pair, although it couldact as both transmitter and receiver for a single transmitter-receiverpair.

In the case of there being more than one transmitter, it is desirablethat the signals transmitted by each transmitter, and their echoes, canbe differentiated from each other. This may be accomplished bytransmitting the signals such that they do not overlap. Alternatively oradditionally, the signals from each transmitter may have a differentbase frequency to each other and/or be differently encoded using asuitable encoding scheme.

Equally the invention is not limited to having just twotransmitter-receiver pairs; in fact any number of pairs could beprovided. Clearly the more pairs provided, the more zones there will bewhich allows more complex and more variety of movements to berecognised. It also means that the zones can be smaller and/or moreclosely spaced which allows smaller movements to be detected. Byappropriate placement of a numerous transmitter-receiver pairs across acontrol surface, particularly such that there are criss-crossingdetection zones to form a grid, embodiments of the invention can allow adegree of two-dimensional location relative the surface to be inferredsuch that basic tracking could be contemplated. Whilst this is withinthe scope of the invention, the Applicant's other applications disclosebetter ways of achieving tracking if that is the goal. Nonetheless it isenvisaged that there may be circumstances where the tracking accuracyrequirement is such that this approach is advantageous over othermethods.

In most aspects of the invention the signals employed are acoustic,preferably ultrasonic. The use of ultrasound is particularlyadvantageous due to its inaudibility to humans and its relatively slowspeed of travel through air, which avoids the need for the highclock-speed equipment needed for speed-of-light timing. This enablesexisting designs for personal electronic equipment, such as mobiletelephones, PDAs and electronic photograph-frames, to be adapted easilyto use the present invention at relatively low cost.

Furthermore the propagation characteristics of ultrasound mean that thetransmitters and receivers may be shallow or even flush-mounted with asurface (for example, a flat-panel display screen) while still beingable to emit and receive sound through a wide range of angles (e.g.potentially a full hemisphere). This is in contrast to a light sourceand sensor (e.g. a camera), which would typically need one or morelenses (e.g. a fish-eye lens) to protrude from the mounting surface inorder to obtain a comparably wide field of emission or view; suchprotrusion can be disadvantageous aesthetically, but also technicallydue to an increased risk of soiling or damage to the lens or otherassociated components.

Another benefit of using ultrasound which the Applicant has appreciatedis that the transducers are flexible in terms of how they can be used.For instance transducers used to implement the present invention mightalso be usable for a fine tracking system, e.g. based on the analysis ofimpulse responses, with appropriate different analysis of the receivedsignals and possibly also different signal transmission. The necessarychanges could though be implemented completely in software, so that agiven device could be used either for recognising movements as describedherein, or for position tracking. With appropriate transmission signalsand analysis, both modes could even be operated simultaneously.

A further important advantage of using ultrasonic transducers in thiscontext, is that they can be used to discern objects from one anotherbased on range, whilst being insensitive to the angle. This means thatan ultrasonic solution—whilst ‘smearing’ objects having similar rangebut different angles—is also ‘robust’ or ‘indifferent’ with respect tothe angle of the moving object relative to the transducer setup. Whendetecting a gesture using a transmitter and a receiver located on thetop and bottom of a screen, respectively for example, the detection of aleft or right gesture is fairly insensitive to how far up on the screenthe hand is moving—i.e. it is insensitive to the angle between thetransmitter, the hand and the tracking surface. It could be zerodegrees, 45 degrees or even 90 degrees. By contrast an infrared systemwill typically have a limited angular opening in order to sharply filterout movements not coming from within a specific angular cone. This conecan be widened, but this results in a ‘range-smearing’, i.e. if therewithin the ‘target zone’ is more than a single reflector, or if thereflector is not a point-reflector, the resulting range read-out is atthe best an ‘average range’ and at the worst, a range not representativeof any point lying within the intended tracking space. This lattereffect is a direct consequence of the speed of light. By comparison therelatively very slow speed of ultrasound can enable meaningful readoutssuitable for gesture detection.

A yet further advantage of the omni-directional nature of ultrasound isthat it gives an ability to recognize gestures beyond the edge of acontrol surface. For instance, the proximity criteria could be used torecognize a right movement gesture above the screen, or an up or downmotion at the side of the screen, effectively increasing the size of theworkspace. Such operation would be impossible using ordinary infra-redsensors.

Each transmit signal might be a continuous signal which would thenpreferably be of non-constant frequency or non-constant amplitude.Alternatively a sequence of discrete signals is transmitted. In a simpleembodiment these discrete signals could each be a single impulse orspike, i.e. approximating a Dirac delta function within the limitationsof the available bandwidth. In other embodiments each of the discretesignals could be composed of a series or train of pulses. This gives abetter signal-to-noise ratio than a single pulse. In other embodimentseach discrete signal could comprise or consist of one or morechirps—i.e. a signal with rising or falling frequency.

In preferred embodiments of the invention the movement recognition isused to control an electronic device. For example each of thepredetermined movements which can be recognised could correspond to adifferent operation of the device.

When viewed from a further aspect, the invention provides amovement-controlled electronic device comprising:

-   -   a control surface;    -   three or more acoustic transducers arranged around a perimeter        of said control surface so as to form at least two        transmitter-receiver pairs in which the transmitter and receiver        of each pair lie across at least a part of the control surface        from each other; and    -   means for controlling an operation of said device in response to        a movement identified from signals detected by both of said        transmitter-receiver pairs.

The three transducers may consist of two transmitters and one receiver,but preferably consist of one transmitter and two receivers. The devicemay comprise further transmitters and/or receivers arranged to actindependently of the aforesaid three transducers, or in cooperation withone or more of them.

Preferably the signals detected by the receivers of the respectivetransmitter-receiver pairs are transmitted at different times.

In some aspects of the invention, an input to a device is obtained byapplying predetermined proximity criteria to reflected signals receivedover two or more channels. However the invention is not restricted tothe use of proximity criteria. Computationally-efficient, gesture-basedinput can also be realised using algorithms other than proximitycriteria.

Thus, from a further aspect, the invention provides a method ofreceiving input to a system through motion of an object, comprising:

-   -   transmitting an acoustic signal and receiving a reflection of        the signal off the object over a first channel comprising at        least one acoustic transmitter and at least one acoustic        receiver;    -   receiving a reflection of an acoustic signal off the object over        a second channel comprising at least one acoustic transmitter        and at least one acoustic receiver;    -   applying a first algorithm to information relating to the        reflection received over the first channel to determine a first        set of information about the location or motion of the object;    -   applying a second algorithm, different from the first algorithm,        to information relating to the reflection received over the        second channel to determine a second set of information about        the motion of the object; and    -   using said first and second sets of information to determine an        input to the system.

The invention extends to apparatus arranged to receive input throughmotion of an object, comprising:

-   -   transmission means arranged to transmit an acoustic signal;    -   receiving means defining first and second channels with the        transmission means, each channel comprising at least one        acoustic transmitter and at least one acoustic receiver, the        receiving means being arranged to receive a reflection of the        acoustic signal off the object over the first channel, and to        receive a reflection of an acoustic signal off the object over        the second channel; and    -   processing means configured to:        -   apply a first algorithm to information relating to the            reflection received over the first channel to determine a            first set of information about the location or motion of the            object;        -   apply a second algorithm, different from the first            algorithm, to information relating to the reflection            received over the second channel to determine a second set            of information about the motion of the object; and        -   use said first and second sets of information to determine            an input to the apparatus.

The invention also extends to a computer software product, and to acarrier bearing the same, configured, when run on a computer, to cause asystem to receive input through motion of an object, comprising:

-   -   instructions for transmitting an acoustic signal;    -   logic arranged to receive information relating to a reflection        of the acoustic signal off the object received over a first        channel comprising at least one acoustic transmitter and at        least one acoustic receiver, and to process this information to        determine a first set of information about the location or        motion of the object;    -   logic arranged to receive information relating to a reflection        of an acoustic signal off the object received over a second        channel comprising at least one acoustic transmitter and at        least one acoustic receiver, and to process this information to        determine a second set of information about the motion of the        object; and    -   logic arranged to use said first and second sets of information        to determine an input to the apparatus.

By making use of two different algorithms over two different channels,efficient recognition of input, e.g. by means of a hand gesture, ispossible. By arranging the transmitter(s) and receiver(s) of thechannels appropriately, it is possible to implement motion-based controlin a manner that is both computationally-efficient and intuitive to use.

The first algorithm may determine information about the motion of theobject, but preferably determines information only about itslocation—for example, that the object satisfies a predeterminedproximity criterion, such as one described previously. The secondalgorithm may determine information about the object's motion—forexample, that it is moving away from the transducers of the secondchannel.

When combined with appropriate placement of the transducers, thisapproach enables the apparatus to determine useful input usingrelatively fast algorithms, without having to track the motion of theobject using a computationally-intense approach of determining asequence of three-dimensional coordinates by calculating theintersection of a number of quadric surfaces, and then trying tointerpret these coordinates to determine whether a particular inputmotion has been performed.

As with other aspects of the invention, there is no need, in this methodof the invention, to assume that the object acts as a point reflector,or that it has a clear front to be tracked. The apparatus can thereforebe robust regarding the shape of the object used, i.e. whether theobject is a hand or a finger, or some other object, and whether itchanges its shape during the input. This stands in contrast to knownmethods based on repeatedly calculating an intersection of quadricsurfaces. In such methods, specific time-of-flight estimates arecomputed for each transmitter-receiver pair of an apparatus. Here, bycontrast, multiple reflecting surfaces on the object (or multiple tapsin an estimated impulse response image) can be considered as a whole;i.e. providing one overall or average position.

Some known tracking approaches track the centroid of an input object.However, the centroid of a complex input object such as hand isdifficult to identify since, as an object changes its perspectiverelative to the microphones or speakers of the sensing system, achanging set of surface points are visible to the system. The centroidis determined as the ‘average’ of the sensed points at any moment intime; however, as the object changes its orientation relative to thesensing system, this centroid must necessarily move relative to what theuser intuitively understands to be the centroid of the input object.Hence, any tracking using a non-penetrating system must necessarily leadto a mismatch between the system's understanding of where the centroidor centre position of the input object is, and where the userinstinctively understands the object to be. A user must thereforecontrol the input object, such as his hand or finger, in a manner thatmight feel unnatural, in order to compensate for such inconsistencies.This is undesirable. However, input approaches made possible by thepresent invention, which are do not require detailed tracking of anobject's centroid, alleviate such problems.

As mentioned, in preferred embodiments the first algorithm determinesinformation relating to the object's location but not to its motion.This provides a particularly computationally-efficient arrangement.

The first algorithm may determine whether the object satisfies apredetermined proximity criterion, such as those described withreference to earlier aspects of the invention. In some embodiments, thefirst algorithm determines whether the object is within a predeterminedregion.

For example, the first algorithm may determine whether the total time offlight of the signal to the object and back to the apparatus is below athreshold time. When a transmitter and receiver of the first channel arespaced apart, this can facilitate determining whether the object iswithin an ellipsoidal region having foci at the transducers.Alternatively, when the transmitter and receiver of the first or secondchannels are located close to one another; for example, adjacent ortouching, this can facilitate determining whether the object is within asphere centred on the transducers. Of course, there may be otherboundary conditions and constraints which place additional constraintson the shape of such a detection region in practice, such as thephysical location of surfaces of the apparatus.

Advantageously, the first algorithm determines whether the object iswithin a predetermined elliptical radius of the transmitter. Preferablya transmitter and receiver of the first channel are spaced apart on theapparatus by more than a quarter, preferably more than a half, of amaximum dimension of the apparatus. By using such an arrangement, thesetransducers can be used to determine whether the object is within anellipsoid that can be everywhere close to the apparatus; i.e. which canbe long relative to the size of the apparatus without needing toprotrude far away from the apparatus. In this way, it can be determinedwhether the object is close to the apparatus anywhere along a lineacross all or part of the apparatus.

The second algorithm may use information relating to signals receivedover the first and second channels to determine information about themotion of the object, but, at least in some embodiments, it uses onlyinformation relating to signals received over the second channel. Thislessens the computational complexity needed to determine an input.

Preferably the second algorithm uses time-of-flight information relatingto signals transmitted at two different times to determine motion of theobject. For example, the second algorithm may compare twotimes-of-flight to determine if the later-received signal traveledfurther than the earlier-received signal; if so, it may be inferred thatthe object is moving away from the transducers of the second channel.

The comparison of times-of-flight may comprise comparing two valuesdirectly, but may alternatively comprise analysing a matrix of signalsor impulse responses, in which responses from successive transmitsignals are placed in consecutive columns or rows. By doing this thematrix can be treated in a similar way to a bitmap image and subjectedto image-processing techniques. These techniques may look for motion by,for example, applying filters that look for a sloping edge to thematrix. Some such approaches are described in WO 2009/147398 by thepresent Applicant.

In some embodiments, the second channel is used to determine thedistance of the object from an origin, independent of the bearing to theobject from the apparatus (typically so long as the object remainswithin a region predetermined by the physical structure of the apparatusand the directional characteristics of the transducers). Since themotion information thus obtained can be represented as a single value,this can be used very effectively to control a one-dimensional input tothe apparatus. It may alternatively or additionally be used to controljust a binary input to the device; e.g. depending on whether the singlevalue is rising or falling.

Such single-valued input can be used very effectively to control, say, agraphical linear slider on a display screen while avoiding any risk ofcursor ‘slaloming’ which would typically arise when trying to control alinear input using two- or three-dimensional tracking approaches. Forexample, a person typically finds it difficult to effect a purely linearmotion of one hand, as would be required if an input mechanism hadseveral degrees of freedom but a linear input were desired (e.g. movingan on-screen cursor in a straight line when it is capable of being movedin two dimensions). Slaloming may also occur when using known trackingapproaches, based on calculating the intersection of quadric surfaces,if there are even small errors in the time-of-flight estimates. Inpractice, such errors are very common, both due to limited bandwidth andto the fact that the object being tracked is never a perfect pointreflector. This is especially significant when the object is not a pointobject, but rather, a cluster of acoustically reflecting surfaces, withno well-defined or well-definable centroid, as previously explained.

In some preferred embodiments a transmitter and receiver of the secondchannel are located close to each other—e.g. within 10 cm or within 5 cmof one another, or within 5% or 3% of the maximum length of theapparatus. In some other embodiments a single transducer acts as bothtransmitter and receiver. In these cases, the second algorithm may beable to determine whether the object is moving radially towards or awayfrom the transducers of the second channel.

Each channel may have its own transmitter and receiver, but in somepreferred embodiments, the second channel shares either a transmitter ora receiver with the first channel. Such sharing can provide a reductionin manufacturing costs. One or both channels may comprise a plurality oftransmitters and/or a plurality of receivers.

In some embodiments, one of the transmitters shares an active component,such as a membrane or piezo crystal surface, with a receiver. Onetransducer may act both as a transmitter and a receiver, either for thesame channel or for different channels. The reflection received on thesecond channel may arise from the acoustic signal transmitted by thetransmitter of the first channel, or from a separate signal.

The first and second sets of information may comprise coordinates or anyother suitable information. However, in some preferred embodiments, theinformation comprises binary flags indicating whether the object is, forexample, within a predetermined region, or is moving away from a set oftransducers. These information sets may be stored in a computer memory,such as RAM or CPU registers.

The determination of an input to the apparatus makes use of the firstand second sets of information. The placement of the transducers on theapparatus, and the type of input to be determined, will affect exactlyhow the information sets are used to determine the input. In oneexample, an input consists of a sweeping gesture of an object, typicallythe user's hand, from left to right across a horizontally-elongate zonein front of the device. This may be used to interact with a virtualvolume slider displayed on the screen of a television set, for example.It is desirable that a determination of the input is made only when theuser's hand is close to the displayed slider, since there may be otherinteractive elements displayed elsewhere on the screen and so as toavoid false detection of other movements occurring elsewhere in theroom. A transmitter is preferably located on the appliance. Thistransmitter may be shared by both channels.

In one specific example, it will be assumed (merely for the purpose ofthe example) that the transmitter is near the left end of a deviceapparatus. A receiver for the first channel may be located near to oradjacent the right end of the zone, while a receiver for the secondchannel may be located near the transmitter at the left end—e.g. within5 cm. By applying a time-of-flight threshold to signal reflectionsreceived over the first channel, the device can determine when an objectenters an elongate ellipsoid (truncated by the front face of the device)of approximately the same length as the zone on the face of the device,and projecting only a relatively small amount from the face of thedevice. Thus the first set of information may comprise a flag as towhether an object is close to the slider or not. Two or more consecutivetime-of-flight measurements of reflections using the second channel maybe made either at the same time or after a positive determination ismade in respect of the first channel. These may then be inspected inorder to determine whether an object is moving towards or away from thetransmitter and receiver of the first channel. Since it is known thatthe object is, at least initially, close to the slider zone, this wouldimply motion of the object from left to right across the zone or viceversa. This assumes that the same object is being detected in bothchannels, which may be a reasonable assumption to make; or, in someembodiments, this may be verified using additional information and/oranalysis, such as considering reflections from other transducers, orperforming a more detailed analysis of the pattern of reflected signalsin both channels.

It may also be advantageous to keep checking periodically that theobject is within the region determined using the first channel while themotion information is captured using the second channel, in order todetect if the user's hand moves too far away from the slider zone andreact appropriately.

If it is determined that the user's hand is close to the slider and thatit is moving from away from the transmitter; i.e. that it is moving fromleft to right, the volume of say a television set may be increased and agraphical element representing the volume level may be moved insynchronisation with motion of the hand. When the movement of the handstops, or it leaves the zone (as detected by the first channel), thevolume level is set at the corresponding position.

Apparatus according to the aspect of the invention set out above mayhave any number of different channels, each with an associatedalgorithm. Some channels may use the same algorithms as each other, orevery channel may have a unique algorithm associated with it. Somechannels may be capable of providing input into more than one algorithm,either running concurrently or at different times. The association oftransducers (transmitters and receivers) with channels may be fixed ormay change over time; for example, it may be changed dynamicallydepending on the type of input to be determined.

In some embodiments, channels are combined in different modes to obtaina particular advantage. Preferably at least one channel is used in‘proximity’ mode, and at least another in ‘radial’ mode; however, othermodes may be combined, such as array modes. For instance, the apparatuscould comprise two closely-spaced, or adjacent receivers and processingmeans arranged to process signals received from these receivers as anarray. It is thereby possible to provide increased directionalsensitivity. Closely-spaced may here mean closer than the wavelength, orhalf the wavelength, of the highest or lowest frequency emitted by thetransmitter; however, it could be greater than this (e.g. in accordancewith the definition set out above).

The apparatus may be arranged to determine a single input type at atime, but is preferably arranged to identify an input from a set of twoor more possible inputs. These may, for example, be gestures differingin style and/or gestures differing in position relative to theapparatus. They could, for example comprise the same gesture performedin reverse.

In some embodiments, the apparatus supports a plurality of applications(e.g. software applications, such as a music player, picture vieweretc.) or interaction modes. One or more of these may temporarily bedesignated an active application or interaction mode. The apparatus maybe configured to receive an input by identifying a gesture from amongsta finite set of gestures, wherein the composition of the set of gesturesdepends on which application or interaction mode is active; e.g. whichapplication is set to receive the input. The set of gestures, referredto hereinafter as the “active gesture set”, will therefore be subset ofall the gestures which the apparatus is capable of recognising. Forexample, an active gesture set may be chosen depending on whichapplication has been selected by a user or by the apparatus to receiveinput, or on which application is running in the foreground, or on whichapplication is controlling the majority of a display surface.

The idea of application-specific gesture sets is new and inventive inits own right, and thus from a further aspect the invention provides amethod of receiving an input from a moving input object, comprising:

-   -   determining that one of a plurality of applications or        interaction modes is active;    -   selecting, in dependence on the active application or        interaction mode, an active gesture set comprising a subset of a        larger set of predetermined gestures;    -   receiving a signal conveying information relating to movement of        the input object; and    -   processing the received signal to identify a gesture from the        active gesture set and thereby determine said input.

This aspect extends to apparatus for receiving an input from a movinginput object, comprising:

-   -   receiving means arranged to receive a signal conveying        information relating to movement of the input object; and    -   processing means configured to:        -   determine that one of a plurality of applications or            interaction modes is active;        -   select, in dependence on the active application or            interaction mode, an active gesture set comprising a subset            of a larger set of predetermined gestures; and        -   process the received signal to identify a gesture from the            active gesture set and thereby determine said input.

This aspect further extends to a computer software product, and acarrier bearing the same, which, when executed on processing means,causes the processing means to:

-   -   determine that one of a plurality of applications or interaction        modes is active;    -   select, in dependence on the active application or interaction        mode, an active gesture set comprising a subset of a larger set        of predetermined gestures; and    -   process the received signal to identify a gesture from the        active gesture set and thereby determine said input.

By restricting the set of recognisable gestures depending on the activeapplication (e.g. a photo viewer application) or interaction mode (e.g.a slide presentation mode, or a movie-viewing mode), it is possible tosupport intuitive gestures that are appropriate for a particularapplication or mode, while at the same time increasing the system'sdetection rate for a particular gesture. In some embodiments, the systemattempts to find the best match between any input movements of the inputobject and the selected gesture set.

By contrast, if only a single, common, gesture set were to be provided,it would have to be shared across all applications. To minimisegesture-misclassification errors, the set would have to contain only arelatively small set of gestures. This would potentially result in thesame gesture being used for different functions in differentapplications, which could be confusing to the user. Furthermore, alimited set of gestures cannot typically support gestures that aretailored to specific applications. If a larger, single gesture set wereprovided, to support a richer range of gestures, it would then sufferincreased misclassification rates, because it is harder for the systemto distinguish between gestures that would need to be more alike. Itcould also lead to user frustration because, for at least someapplications, there will be redundant gestures which do not perform anyfunction for that application.

An application or interaction mode may have been activated by a humanuser, for example, by clicking an icon, or by the apparatus, forexample, an alarm clock application may become active when apredetermined time is reached. A default application or interaction modemay be activated automatically under certain conditions, such as whenthe device is first powered on. Preferably, the identified gesture isused to control a function of the active application or a function ofthe apparatus defined by the active interaction mode.

The input object may be a stylus, a fingertip, a hand, or any othersuitable object. The gestures may be of any suitable type. An examplegesture might be the input object being swept from left to right past afront face (e.g. a display panel) of the apparatus. The gesture wouldnot typically require contact between the input object and the apparatus(i.e. it is a touchless gesture), although contact is not excluded.

Another example of a gesture is the input object being moved directlytowards the apparatus, being stopped short of contacting the apparatus,and being held static in the stopped position for a predetermined time.This might be affected by a user pushing the palm of his hand in linetowards a display screen, and holding it a distance away from thescreen, much like a “stop” hand signal. This gesture could be used, forexample, with a slide-show application to prevent an automatictransition from one slide to the next. The current slide may be pausedfor as long as the user maintains his hand in the stationary position,with the slide show recommencing once his hand is removed.

Another possible gesture is a motion of the object along a path directlytowards a part of the apparatus, then immediately away again alongapproximately the same path, in a “tapping” motion. This gesture mightbe used to select an icon on a display screen.

Further possible gestures include up, down, left or right sweepingmovements, e.g. past a display surface.

Such gesture may be tested for, or identified, by any suitable means.Optical, acoustic, radio-frequency, or any other appropriate mechanismmight be used. In some embodiments, method steps described elsewhere inthis specification could be employed to identify a gesture.

It will be appreciated that, for the recognition of certain inputs, theplacement of the transducers relative to one another and to theapparatus can be important. For example, when the apparatus comprises adisplay screen or other interaction surface, the transducers may belocated adjacent the screen or surface in specific ways to supportparticular graphical user-interface elements.

In one aspect, the invention provides apparatus for receiving input froma moving object, comprising a first acoustic transmitter, a firstacoustic receiver, a third acoustic transducer, and processing means,the apparatus being arranged to:

-   -   transmit at least one acoustic signal from said transmitter;    -   receive at least two acoustic reflections off the object; and    -   process the received signals so as to determine an input due to        the motion of the object,        wherein the distance separating said transmitter and said third        transducer is at least twice the distance separating said        transmitter and said receiver.

The third acoustic transducer may be a transmitter or a receiver. Afirst channel may be defined comprising the third transducer and thefirst transmitter or receiver. Due to the relatively large separation ofthe elements of this first channel, it is well suited to determining thepresence of an object in an ellipsoid region, as described previously,although the region may have another shape. A second channel may bedefined comprising the first transmitter and receiver. Their relativelysmall separation facilitates determining movement away from or towardsthe transmitter and receiver, largely independent of the direction ofsuch movement (since a given time-of-flight will define an approximatelyspherical surface).

This arrangement of transducers, which facilitates the defining of arelatively elongate ellipsoidal region and also a relatively sphericalregion, has been found to give particularly advantageous performancewhen it is desired to receive approximately linear input movements. Thecomputational requirements to determine the presence of an object insidean ellipse, and to determine radial movement away from or towards anorigin, are relatively low compared with three-dimensional trackingrequiring the real-time calculation of intersections of multiple quadricsurfaces. This provides substantial advantages in terms of reliabilityand cost of manufacture. As previously mentioned, methods of theinvention may also be more robust when complex shapes, such as a humanhand, are used as the input object, since they do not need to try toresolve the precise coordinates of a centroid of the object.

Preferably the separation distance between the first transmitter and thethird transducer is at least ten times the distance separating the firsttransmitter and the first receiver. Preferably the first transmitter andthe first receiver are adjacent or touching (as defined hereinabove).They may in some embodiments share some components or be the samephysical transducer.

Preferably the three transducers define an input zone with the firsttransmitter and first receiver at one edge or face of the zone and thethird transducer at an opposite edge or face, with the zone lyingbetween these edges or faces. The processing means may then beconfigured to receive inputs at least or only when the object is movingwithin the input zone.

In all the above aspects, the transmitted signal may be audible but ispreferably ultrasonic; e.g. having frequencies greater than 20 kHz,especially greater than 30 kHz. In some embodiments the frequency mightbe in the range 35-45 kHz. In other embodiments a higher frequency thanthis could be used. Thus in one set of embodiments the frequency isgreater than 50 Hz or even greater than 100 kHz—e.g. between 100 and 200kHz. The transmitters could be controlled to transmit continuous signalsor discrete impulses. The signal may comprise a single frequency, or maycomprise a plurality of frequencies. It may be of any suitable type; forexample, a pulse, a chirp, a train of pulses, a succession of chirps;and may be discrete or continuous or continual.

Various features have been described with reference to one of moredifferent aspects of the invention. These features are not limited onlyto those aspects, but, where appropriate, may also be features of any ofthe other aspects. Wherever a method according to the invention isdescribed herein, it should be understood that the invention extends tosuitable apparatus configured to implement the method, and to a softwareproduct, and a carrier bearing the same, which, when run on dataprocessing means, causes the processing means to carry out steps of themethod.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a perspective drawing of a gesture-responsive display unitaccording to one aspect of the invention;

FIG. 2 is a figurative plan view of the display unit showing progressivemovement of a hand past the unit;

FIG. 3 is a perspective drawing of a different gesture-responsivedisplay unit embodying the invention;

FIG. 4 a is a schematic horizontal cross-section through a device inaccordance with a different aspect of the invention;

FIG. 4 b is a schematic front view of the same device;

FIG. 5 is a figurative plan view of an interaction with the device;

FIG. 6 is a perspective view of the front of a television set embodyingthe invention;

FIG. 7 is a perspective view of the front of a different television setembodying the invention;

FIG. 8 is a perspective view looking down on a laptop embodying theinvention; and

FIG. 9 is a perspective view looking down on a different laptop inaccordance with another aspect of the invention.

FIG. 10 is a flow chart of a method according to the invention; and

FIG. 11 is a schematic diagram showing an apparatus embodying theinvention.

DETAILED DESCRIPTION OF CERTAIN INVENTIVE EMBODIMENTS

Certain embodiments of the invention will now be described, by way ofexample only, with reference to the accompanying drawings.

FIG. 1 shows a gesture-responsive display unit 2 having a stand to therear (not visible) which keeps it fixed at a slight incline to thevertical. It comprises a flat-panel LCD screen 4, suitable for showingdigital photographs, mounted in a frame 6. The top horizontal portion ofthe frame has, centrally flush-mounted therein, an ultrasoundtransmitter 8. The bottom horizontal portion of the frame has,flush-mounted therein, a left ultrasound receiver 10 and a rightultrasound receiver 12, located respectively to the left and right ofcentre.

In use the transmitter 8 is arranged to transmit ultrasonic pulses atperiodic intervals (e.g. every 1/100 seconds). The pulses might all bethe same, or could, for example, alternate between a higher and a lowerfrequency. Each pulse lasts for example 10 micro-seconds. The left andright receivers 10, 12 each receive the signals. An inbuilt processorand associated circuitry coordinates the timing of the transmit signals,and also decodes the received signals and stores them in a memory bufferto allow subsequent analysis for determining the time at which signalswere received.

As will be explained in more detail below, the display unit 2 isarranged to be responsive to a gesture of a human hand 14 passing infront of the LCD screen 4, e.g. from right to left, with the nearestpart of the hand to the screen (typically the fingertips) being at arange of approximately 1 cm to 20 cm from the surface of the screen. Inparticular, it is arranged to respond to the gesture be moving fromdisplaying one photograph to the next, from an ordered album ofphotographs. A visual animation of the new photograph sliding in fromthe side over the top of the preceding photograph can be used to enhancethe user experience.

The detection of this gesture is accomplished through analysis of thesignals received after reflection from the hand 14. In particular,depending on the position of the hand 14, a transmitted signal followstwo separate paths 16, 18 to the hand and away from it again afterreflection to the left receiver 10 and right receiver 12 respectively.It may be observed in the diagrams that the reflections are shown ascoming from two different fingers. This illustrates an advantageinherent in at least preferred embodiments of the invention whereby itis not necessary to track the movement of a specific part of theobject—e.g. a specific finger tip. The ability to avoid having to‘focus’ on a particular part of the hand makes the system more reliableand less complex than known single-range measurement systems such as afinger-tip based ultrasonic tracking system or a general infra-redtracking system.

The details are explained with reference to FIG. 2, which shows anapproximate plan view of the display unit 2 of FIG. 1. Although not infact be visible from above, the positions of the transmitter 8 andreceivers 10, 12 have been shown for convenience. Four successivepositions of the hand 14 in its journey from right to left in front ofthe display unit 4 are illustrated 14 a, 14 b, 14 c, 14 d.

Each periodic transmit signals is received at the receivers 10, 12 alonga direct path from the transmitter 8 to the receiver. However since thegeometry of the transducers is fixed, this is easily predicted and canbe disregarded. Reflections or echoes of the signal from objectssituated in front of the screen will typically follow shortly after thedirect path. The total time of flight from the transmitter 8, via anobject creating an echo, to a receiver 10, 12 conveys informationconcerning the position of the object in three dimensional spacerelative to the unit 4. In particular, receipt of an echo within acertain time determines that at least a part of the object giving riseto that echo is situated within an ellipsoid, having the receive andtransmitter as foci, where the size of the ellipsoid is a function ofthe time. The fact that an echo must be sufficiently strong to be ableto detect means that any recorded time of flight will also depend theobject's reflectiveness to ultrasound, which will be influenced by theobject's size, shape and composition.

In order to detect movement of the hand 14 from right to left, it issufficient in the present embodiment to determine that the hand islocated at a first position 14 b in front and to the right of the screenat a first point in time, and is located at a second position 14 c infront and central or to the left of the screen at a second point intime. To ensure that a gesture is detected only when performed at aminimum speed, a maximum allowed time threshold is set between the firstand second points in time in order for the gesture to be registered.

Virtual left and right proximity zones 24, 26 can be imagined in respectof the left 10 and right 12 receivers respectively. These are defined bya maximum time-of-flight threshold in which an echo can be detectedafter transmission of a tone from the transmitter 8. In order to ensurethat an echo originates from the user's hand 14, and not from somesmaller object, such as an insect flying in front of the screen, aminimum energy threshold is set for the received signal in order toregister a qualifying echo.

Considering the right receiver 12 first: after a signal is sent from thetransmitter 8, a direct signal will be received at the right receiver12. The direct-path signal is of no interest and is ignored or removedwith a suitable filtering or subtraction scheme. Immediately after thedirect-path signal has been received, a sliding time window is opened tolisten for echoes. A rolling sum of the energy received during the timewindow is calculated, and when this exceeds a predetermined threshold, ahand detection event is registered. The sliding window is only kept openfor a predetermined length of time, corresponding to a predeterminedproximity criterion or the size of the aforementioned ellipsoid. Afterthis time, if no echo is registered for that transmit signal, noreceived signals are recorded until the next signal is transmitted fromthe transmitter 8, at which point the process repeats.

The same process occurs in respect of the left receiver 10.

If a detection event is recorded in respect of either channel 8, 10 or8, 12, further analysis is carried out to determine which occurredfirst, and whether a detection event was recorded in respect of theother channel resulting from a subsequently transmitted signal.

If a hand detection event is first recorded for the right channel 8,12(e.g. when the hand is at location 14 b) and another hand detectionevent was subsequently detected by the left channel 8,10 (e.g. when thehand is at location 14 c or 14 d), as the result of an echo from asubsequent signal, within a predetermined time after the right-receiverdetection event, an identification of a right-to-left gesture is madeand the screen 4 is updated accordingly.

A left-to-right gesture may be identified if the order of detectionevents is reversed (i.e. the left channel 8,10 detects first, followedby the right channel 8,12).

In order to reduce the risk of false detections, a time period may beprovided after a gesture has been successfully identified before thesystem is ready to identify a further gesture.

Considering the scenario of FIG. 2, the hand 14 is initially at aposition 14 a out of detection range of either transmitter-receiverpair. As the hand starts to move in front of the right side of thedisplay unit 2, it reaches a position 14 b where enough of the hand(e.g. two fingertips) are within the detection zone 26 of the rightchannel 8,12 that a signal transmitted from the transmitter 8 isreflected from the hand 14 with sufficient energy that the rightreceiver 12 receives, within the sliding time window, energy exceedingthe predetermined threshold. It therefore registers a first handdetection event.

As the hand 14 continues moving leftward, it reaches a position 14 cwhere it is sufficiently-far within the detection zone 24 of the leftchannel 8,10 that a transmit signal (subsequent to the tone that wasresponsible for the first detection event) reflects off the hand withsufficient energy that a second detection event is recorded for the leftreceiver 10. Since this second event occurred after the first event, butstill within the predetermined reset time period, a right-to-leftgesture is flagged as having occurred and a signal is sent to theappropriate local or remote processing system for a suitable response tothe gesture to be effected, such as changing the display 4 to show thenext photograph in the album.

Although, for clarity, the operation of the device has been described asif the processing happens in “real time”, in fact, the signals receivedfrom the receivers 10, 12 are typically stored in a temporary storagemedium such as RAM and then processed slower than real time. Inparticular, because hand detection events for successivetransmitter-receiver pairs may occur in too rapid succession forreal-time processing to reliably register both, if a hand detectionevent occurs for one channel, further processing of the signals for thatchannel may be halted and an analysis of the stored received signals forthe other channel commenced in order to check, retrospectively, whethera subsequent hand detection event occurred in respect of that channelalso.

In order to provide better resistance from background noise, instead offlat pulses being transmitted, chirped signals may be used. In oneembodiment each signal comprises a rising tone followed immediately by afalling tone. The received signals are then subject to a de-chirpoperation, corresponding to the shape of the transmitted chirp; in thisway background noise can be effectively filtered out. However thisrequires more sophisticated signal processing which may not be availableon some devices or may be too power-hungry.

FIG. 3 shows a second embodiments of a display unit 30 having an LCDscreen 32 and a border 34. It operates in a somewhat similar fashion tothe previous embodiment, but instead of having only one transmitter 8and two receivers 10, 12, the display unit 30 has three transmitters 36a, 36 b, 36 c horizontally spaced along the top of the frame 34; threereceivers 38 a, 38 b, 38 c horizontally spaced along the bottom of theframe 34; three transmitters 40 a, 40 b, 40 c vertically spaced down theleft side of the frame 34; and three receivers 42 a, 42 b, 42 c(obscured) vertically spaced down the right side of the frame 34. Eachtransmitter cooperates with its opposite neighbour, located horizontallyor vertically across the screen 32 from it, to form a channel. In thisway, six different detection zones may be defined. These may overlapwith each other, as in the previous embodiment, but could be arranged tobe distinct, by setting the upper bound on the maximum time-of-flightfor each pair to be sufficiently low (this was not possible in theprevious embodiment because the transmitter 8 served twotransmitter-receiver pairs, and was therefore a focus for two differentellipsoids which must necessarily therefore have some overlap).

Having three horizontal detection zones and three vertical detectionzones, each spanning the full width of the screen 32, effectivelyenables a 3×3 grid of nine regions to be established over the surface ofthe screen 32. In this way, more complex gestures such as circlingmovements or diagonal sweeps can be identified, based on the order inwhich detection events are recorded for the various transmitter-receiverpairs.

If some of the transmitters and receivers are reused for multiplechannels, even more complex detection zones may be established.

FIGS. 4 a and 4 b show different views of a device 50 in accordance withanother aspect of the invention. The device 50, which may be atelevision screen, computer monitor, portable music player, mobilephone, or the like, has a display screen 52 surrounded by a frame 54.The screen 52 is displaying a graphical volume slider 53. A mark 55 onthe slider corresponds to the current volume level for audible soundemitted from the device 50.

Mounted in the frame 54 to the left of the screen 52 are an ultrasoundtransmitter 56 and a left ultrasound receiver 58. To the right of thescreen 52 is a right ultrasound receiver 60.

FIG. 5 shows use of the device 50 by a user to increase the volume. Acontinuous or periodic acoustic signal is transmitted from thetransmitter 56. This may be a pulse or tone, or preferably a chirp.Electronic circuitry connected to the right receiver 60 monitors thereceived signals to identify reflections of the signal received by theright receiver 60 within a maximum time-of-flight bound. It may beaccomplished using any appropriate technique; for example, by opening atime window immediately after transmission of a signal by thetransmitter 56, applying a de-chirp operation to received signals,filtering out a direct-path signal from the transmitter 56 to thereceiver 60, integrating received energy in the de-chirped signal over asliding time sub-window, determining if greater than a threshold energyis received at any point during the sub-window, and closing the maintime window when the maximum time-of-flight bound elapses. Other methodsare possible; however this has the advantage of requiring relativelylittle computational processing effort.

The acoustic signal preferably repeats over a time frame somewhat longerthan the time taken for sound to travel twice the width of the device50.

The maximum time-of-flight boundary effectively defines an ellipsoidregion 62. If a suitably reflective object enters this region 62, thedevice 50 will detect this.

The graphical slider may be permanently displayed on the screen, but itpreferably appears at this point; i.e. when the user's fingertip 14 a isdetected near the relevant part of the screen 52.

Also at this point, electronic circuitry connected to the left receiver58 starts monitoring the received signals to identify reflectionsreceived by the left receiver 58. By any appropriate means, itdetermines whether successive reflections from the user's fingertip 14 aare progressively taking longer to arrive or less time to arrive. Thismay be accomplished by recording a series of time-of-flightmeasurements, a measurement being recorded whenever the received energyfrom a reflected signals exceeds a threshold within a sliding timesub-window, as above. By comparing adjacent measurements, or byanalysing a trend across several measurements, the circuitry candetermine whether the fingertip 14 a is moving radially towards oroutwards from the left side of the frame 54 in an approximatelyspherical manner. An example spherical region 64 is shown in FIG. 5.

Alternatively, this radial channel may compute an approximate positionof the hand 14 or fingertip 14 a, which could be derived as an averageposition of multiple reflective points, or as multiple ‘taps’ in animpulse response. Advantageously this can mean that the user does notnecessarily have to extend his/her finger in order to control the volumeslider 53. Instead, multiple fingers, a fist, or a scooping hand couldbe used, which can provide a more natural feel to the user.

By itself, i.e. not in conjunction with the ellipsoid channel, thisradial position would only give radial information, i.e. it could notpoint to where in space the finger is, just how far away it is. Inconjunction with the first channel, however, the device 50 can calculatehow far away within the proximity zone it is, which translates tobecoming the x-position along the surface and between the elements.

While this motion determination is being made, the circuitry connectedto the right receiver 60 continues to determine whether the user'sfingertip 14 a remains within the ellipsoid region 14 a, close to thegraphical slider 53.

As the user moves his hand 14 in the direction indicated by the arrow inFIG. 5, the control circuitry driving the display screen 52 moves themark 55 on the slider 53 progressively rightwards, and the volume of anyaudible sound being output from the device 50 is also increased.

The device 50 may be configured to identify crudely anysubstantially-large left-to-right sweep as an input to the device. Thismay be accomplished by comparing just two time-of-flight measurementsfrom the left receiver 58. The device 50 may then increase the volume bya fixed increment, with a number of repeated sweeps being required toincrease the volume by a large amount.

Alternatively, the device 50 may monitor the movement of the fingertip14 a in greater detail, for example by considering several successivetime-of-flight measurements from the left receiver 58. It may then bepossible to set the volume by moving the fingertip 14 a from left toright and stopping the movement around the desired position for the mark55 on the slider 53. The device 50 may fix the volume level whencessation of movement of the fingertip 14 a from left to right isdetected, or when the fingertip 14 a is removed from the ellipsoidregion 62 (e.g. along a path roughly perpendicular to the face of thedisplay screen 52).

A right-to-left movement or series of movements may be detectedsimilarly by determining an approaching motion of the fingertip 14 atowards the transmitter 56.

FIG. 6 shows an embodiment in the form of a television set 70. Beneaththe screen 72, it has a loudspeaker panel 74, containing one or moreloudspeakers 75 behind a mesh grille or fabric cover (not shown). Theseloudspeakers 75 are used conventionally for reproducing audible sound.This same panel region 74 can advantageously also be used as a sensingzone. This allows the transducers for the input mechanism to be hiddenfrom view behind the same mesh as protects the speaker elements 75.Using this region 74 as a zone for gesture-based inputs also allows suchinputs to be made without obscuring the screen 72, which may bedesirable in certain situations.

In the loudspeaker panel 74 there is an ultrasound transmitter 76situated near the right edge of the television set 70; a firstultrasound receiver 78, situated close to the transmitter 76 in order toform a first, ‘radial’ channel; and a second ultrasound receiver 80,situated relatively far from the transmitter 76 near the left edge ofthe set 70 in order to form a second, elongate or ‘ellipsoid’ channel,substantially spanning the width of the television set 70. These can beoperated to provide a volume input as already described. Alternativelyor additionally, they may facilitate other inputs, such as channel upand down.

The television set 82 of FIG. 7 is very similar to that of FIG. 6,except for the addition of two further acoustic receivers 94, 96situated more centrally in the loudspeaker panel 86. This increase inthe number of receiving elements enables additional ‘radial’ channelsand ‘ellipsoid’ or ‘proximity’ channels to be established and used incombination, or interchangeably, to increase the precision or thereliability of the system.

FIG. 8 shows a laptop 98 comprising a display panel 100 connected by ahinge to a base panel 102 comprising a keyboard 104. It has a rightspeaker panel 103 to the right of the keyboard 104 and a left speakerpanel 105 to the left of the keyboard 104. These house conventionalaudio speakers (not shown) for outputting stereo sound. An ultrasoundtransmitting element 106 and a left receiving element 108 are embeddedin the left speaker panel 105. A right receiving element 110 is embeddedin the right speak panel 103. In this way, an x-positioning zone can becreated across and above the keyboard surface 104; i.e. by appropriateoperation of the ultrasound transducers, movements having a linearcomponent from one side of the keyboard 104 towards the other side canbe detected.

FIG. 9 shows a laptop 112 outwardly similar to the laptop of FIG. 8, butoperating in a manner more closely related to the gesture-responsivedisplay unit 2 of FIG. 1. It has a left ultrasound transmitter 120 and aright ultrasound transmitter 122 mounted on a base panel 116, one oneither side of the keyboard 118. The display panel 114 has, mountedalong its upper edge, three ultrasound receivers; a left receiver 124near the top left corner, a central receiver 126 near the middle of theupper edge, and a right receiver 128 near the top right corner.

In use, the left transmitter 120 may cooperate with right and centralreceivers 124, 126 to define two virtual proximity zones as describedabove with reference to

FIG. 2. The right transmitter 122 may cooperate with central and rightreceivers 126, 128 to define a further two virtual proximity zones.Other combinations are possible instead or additionally. Side-to-sidehand motions across the face of the display screen 114 may thus bedetected using methods of the invention already described.

The ultrasound transmitters 120, 122 and ultrasound receivers 124, 126,128 are thus located on different panels 116, 114 of the laptop,connected by a hinge. In normal use, when the laptop 112 is open, thesepanels will be very approximately at right angles to one another. The isadvantageous in that the transmitters 120, 122 are physically angledtowards the receivers 124, 126, 128 across the face of the display panel114. This enables the use of directional transmitters for recognising amovement of an object in the vicinity of the display panel 114, ratherthan needing omnidirectional transmitters as might be the case fortransmitters and receivers mounted on a common panel. Directionalultrasound transmitter are typically cheaper than omnidirectionaltransmitters and can be driven with less power for the same level ofreceived energy at the receivers 124, 126, 128 along reflected pathsclose to the screen 114.

Having the ultrasound transmitters 120, 122 mounted on the base panel116 is also advantageous in that they can be hidden under the kind ofmesh often used to hide audio speaker elements in existing laptopdesigns, as is the case also for the laptop 98 shown in FIG. 8.

FIG. 10 illustrates a method of recognizing a movement of an object asone of a set of one or more possible predetermined movements, comprisinga step 130 of continuously or repeatedly transmitting signals between atleast two transmitter-receiver pairs, the signals being reflected fromthe object to produce reflected signals; a step 132 of recording a firstevent for a first one of the transmitter-receiver pairs when a reflectedsignal meets a first predetermined proximity criterion; a step 134 ofrecording a second event for a second one of the transmitter-receiverpairs when, after the first event, a subsequent reflected signal meetsthe first predetermined proximity criterion or a second predeterminedproximity criterion; and a step 136 of using the first and second eventsto identify the movement as one or more of the predetermined movements.

FIG. 11 shows an apparatus 138 for recognizing a movement of an object.It comprises a display 142, similar to displays, such as display 4,described elsewhere herein. The apparatus 138 comprises twotransmitter-receiver pairs. The first pair comprises a first transmitter144 and a first receiver 146. The second pair comprises a firsttransmitter 148 and a first receiver 150. The apparatus 138 alsocomprises an electronic controller 152 arranged to: continuously orrepeatedly transmit signals between the at least twotransmitter-receiver pairs, the signals being reflected from said objectto produce reflected signals; record a first event for a first one ofsaid transmitter-receiver pairs when the reflected signal meets a firstpredetermined proximity criterion; and record a second event for asecond one of said transmitter-receiver pairs when, after the firstevent, a subsequent reflected signal meets the first or a secondpredetermined proximity criterion. The apparatus 138 also comprises anelectronic processor 154 arranged to use the first and second events toidentify the movement as the or one of the predetermined movements.

The invention claimed is:
 1. A method of recognizing a movement of anobject as one of a set of one or more possible predetermined movements,the method comprising: continuously or repeatedly transmitting acousticsignals between at least two transmitter-receiver pairs, said signalsbeing reflected from an object to produce reflected signals; analyzingsignals received at the receiver of a first one of saidtransmitter-receiver pairs to determine whether one or more reflectedsignals, received at the receiver, meets a first predetermined proximitycriterion, the first predetermined proximity criterion comprising thereflected signal or signals having a time of flight, or average time offlight, from the transmitter to the receiver of the transmitter-receiverpair, via the object, that is within a first predetermined maximum timeof flight, and recording a first event for the first one of saidtransmitter-receiver pairs when one or more reflected signal meets thefirst predetermined proximity criterion; analyzing signals received atthe receiver of a second one of said transmitter-receiver pairs todetermine whether one or more reflected signals, received at thereceiver, meets a second predetermined proximity criterion, the secondpredetermined proximity criterion comprising the reflected signal orsignals having a time of flight, or average time of flight, from thetransmitter to the receiver of the transmitter-receiver pair, via theobject, that is within a second predetermined maximum time of flight,and recording a second event for the second one of saidtransmitter-receiver pairs when, after the first event, one or moresubsequent reflected signals meets the second predetermined proximitycriterion; and using said first and second events to identify a movementof the object as one of a set of one or more possible predeterminedmovements.
 2. The method of claim 1, wherein a control surface separatesthe transmitter and receiver of each pair.
 3. The method of claim 1,wherein one or both of the first and second predetermined proximitycriteria additionally comprises each of the one or more reflectedsignals containing at least a minimum energy reflected from the object.4. The method of claim 1, wherein the first event is recorded after thereflected signals corresponding to both the first and the second eventshave been received.
 5. The method of claim 1, further comprisingdetermining quantitative measures of the proximity of the object to eachtransmitter-receiver pair.
 6. The method of claim 5, further comprisingrecording the first or second events for the transmitter-receiver pairthat is closest to the object.
 7. The method of claim 1, furthercomprising using the identity of the transmitter-receiver pairscorresponding to the first and second events, and the order in which theevents occurred, to identify said movement of the object as one of theset of one or more possible predetermined movements.
 8. The method ofclaim 1, further comprising using the time interval between the firstand second events to identify said movement of the object as one of theset of one or more possible predetermined movements.
 9. The method ofclaim 1, wherein the acoustic signals are acoustic signals.
 10. Themethod of claim 1, wherein the second predetermined proximity criterionis the same as the first predetermined proximity criterion.
 11. Themethod of claim 1, wherein the first or second predetermined proximitycriterion comprises the reflected signal or signals having a time offlight, or average time of flight, from the transmitter to the receiverof the transmitter-receiver pair, via the object, that is within apredetermined range interval, the method further comprising detectingwhether there is an object present within the predetermined rangeinterval by applying an energy detector, or an edge detector, or anenvelope shape detector to corresponding portions of one or more signalsreceived by the first or second transmitter-receiver pairs, or tocorresponding portions of impulse responses calculated from signalsreceived by the first or second transmitter-receiver pairs.
 12. Anapparatus for recognizing a movement of an object as one of a set of oneor more possible predetermined movements, comprising: at least twotransmitter-receiver pairs; electronic controller arranged to:continuously or repeatedly transmit acoustic signals between thetransmitter-receiver pairs, the signals being reflected from an objectto produce reflected signals; analyze signals received at the receiverof a first one of said transmitter-receiver pairs to determine whetherone or more reflected signals, received at the receiver, meets a firstpredetermined proximity criterion, the first predetermined proximitycriterion comprising the reflected signal or signals having a time offlight, or average time of flight, from the transmitter to the receiverof the transmitter-receiver pair, via the object, that is within a firstpredetermined maximum time of flight, and to record a first event forthe first one of said transmitter-receiver pairs when one or morereflected signal meets the first predetermined proximity criterion; andanalyze signals received at the receiver of a second one of saidtransmitter-receiver pairs to determine whether one or more reflectedsignals, received at the receiver, meets a second predeterminedproximity criterion, the second predetermined proximity criterioncomprising the reflected signal or signals having a time of flight, oraverage time of flight, from the transmitter to the receiver of thetransmitter-receiver pair, via the object, that is within a secondpredetermined maximum time of flight, and to record a second event forthe second one of said transmitter-receiver pairs when, after the firstevent, one or more subsequent reflected signals meets the secondpredetermined proximity criterion; and an electronic processor arrangedto use the first and second events to identify a movement of the objectas one of a set of one or more possible predetermined movements.
 13. Theapparatus of claim 12, wherein for each transmitter-receiver pair, thetransmitter and receiver of the pair are separated from one anotheracross at least a part of a control surface.
 14. The apparatus of claim13, wherein the transmitter of each transmitter-receiver pair is on anopposite side of the control surface to the paired receiver.
 15. Theapparatus of claim 13, wherein the control surface comprises a displayfor an electronic device.
 16. The apparatus of claim 13, wherein thecontrol surface is planar and wherein each transmitter-receiver paircomprises a transducer which is mounted flush with or below the plane ofthe control surface.
 17. The apparatus of claim 12, wherein the acousticsignals are acoustic signals.
 18. The apparatus of claim 12, wherein thesecond predetermined proximity criterion is the same as the firstpredetermined proximity criterion.
 19. The apparatus of claim 12,wherein the first or second predetermined proximity criterion comprisesthe reflected signal or signals having a time of flight, or average timeof flight, from the transmitter to the receiver of thetransmitter-receiver pair, via the object, that is within apredetermined range interval, and the electronic controller beingfurther arranged to detect whether there is an object present within thepredetermined range interval by applying an energy detector, or an edgedetector, or an envelope shape detector to corresponding portions of oneor more signals received by the first or second transmitter-receiverpairs, or to corresponding portions of impulse responses calculated fromsignals received by the first or second transmitter-receiver pairs. 20.A non-transitory computer software product configured, when executed ona computer, to recognize a movement of an object as one of a set of oneor more possible predetermined movements, the software performing themethod comprising: receiving data resulting from continuously orrepeatedly transmitting acoustic signals between at least twotransmitter-receiver pairs, the signals being reflected from an objectto produce reflected signals; analyzing signals received at the receiverof a first one of said transmitter-receiver pairs to determine whetherone or more reflected signals, received at the receiver, meets a firstpredetermined proximity criterion, the first predetetermined proximitycriterion comprising the reflected signal or signals having a time offlight, or average time of flight, from the transmitter to the receiverof the transmitter-receiver pair, via the object, that is within a firstpredetermined maximum time of flight, and recording a first event forthe first one of said transmitter- receiver pairs if the data representsone or more reflected signals meeting the first predetermined proximitycriterion; analyzing signals received at the receiver of a second one ofsaid transmitter-receiver pairs to determine whether one or morereflected signals, received at the receiver, meets a secondpredetermined proximity criterion, the second predetermined proximitycriterion comprising the reflected signal or signals having a time offlight, or average time of flight, from the transmitter to the receiverof the transmitter-receiver pair, via the object, that is within asecond predetermined maximum time of flight, and recording a secondevent for the second one of said transmitter-receiver pairs if, afterthe first event, the data represents a subsequent one or more reflectedsignals meeting the second predetermined proximity criterion; and usingthe first and second events to identify a movement of the object as oneof a set of one or more possible predetermined movements.
 21. Thenon-transitory computer software product of claim 20, wherein the secondpredetermined proximity criterion is the same as the first predeterminedproximity criterion.